Practical application of one-pass Viterbi algorithm in tokenization and part-of-speech tagging

Sentence word segmentation and Part-OfSpeech (POS) tagging are common preprocessing tasks for many Natural Language Processing (NLP) applications. This paper presents a practical application for POS tagging and segmentation disambiguation using an extension of the one-pass Viterbi algorithm called Viterbi-N. We introduce the internals of the developed system, which is based on lattices and a stochastic model built using second order Hidden Markov Models (HMMs). Also, we present the results of an evaluation process and the analysis of the error cases. The results achieved suggest that the Viterbi-N algorithm applied on lattices allows POS tagging and segmentation disambiguation to be accomplished in a common process. Although the tests were done for the Galician language, the solution proposed could be easily exported to other